Skip to content

Profiler, run-history timeline, secrets, webhook + email triggers, embedder fixes#184

Merged
JE-Chen merged 12 commits intomainfrom
dev
Apr 28, 2026
Merged

Profiler, run-history timeline, secrets, webhook + email triggers, embedder fixes#184
JE-Chen merged 12 commits intomainfrom
dev

Conversation

@JE-Chen
Copy link
Copy Markdown
Member

@JE-Chen JE-Chen commented Apr 28, 2026

Summary

Five new headless-first features and two embedder-friendliness fixes built on top of the existing executor/trigger/run-history infrastructure.

Features

  • Profilerdefault_profiler + AC_profiler_* commands record per-action wall-clock durations; opt-in (zero overhead when disabled). Profiler GUI tab shows hot spots with calls / total / avg / share.
  • Run history timeline — Gantt-style strip in the Run History tab plus an inline failure-screenshot preview pane. Selection syncs both ways between table and strip.
  • Secret manager — Fernet-encrypted JSON vault under ~/.je_auto_control/secrets/ with PBKDF2-HMAC-SHA256 (600k iterations) key derivation. Action scripts reference entries via ${secrets.NAME} placeholders so plaintext never enters the variable scope or script JSON. New Secrets GUI tab + AC_secret_* commands.
  • Webhook triggerhttp.server-backed HTTP push trigger; configurable path / methods / bearer token; request method, path, query, headers, body, and parsed JSON are seeded into the variable scope as ${webhook.body} etc. New Webhooks GUI tab + AC_webhook_* commands.
  • IMAP email trigger — Poll-based watcher; subject / sender / body / message-id / uid available as ${email.subject} etc. UID-deduplicated and optionally marks \Seen. New Email Triggers GUI tab + AC_email_trigger_* commands.

Embedder fixes (PyBreeze regression)

  • gui/__init__.py no longer eagerly imports main_window; the launcher's PySide6 + WebRTC stack is loaded lazily inside start_autocontrol_gui().
  • gui/main_widget.py wraps the RemoteDesktopTab import in try/except and substitutes a placeholder tab with install instructions when the optional webrtc extra (aiortc + PyAV) is unavailable. Embedders that import from je_auto_control.gui.main_widget import AutoControlGUIWidget on a base install no longer fail at import time.

CLAUDE.md compliance

  • All five features ship a headless API in utils/, an AC_* executor command, a re-export from je_auto_control/__init__.py, and a thin GUI tab.
  • import je_auto_control stays Qt-free (verified: no PySide6 in sys.modules).
  • 56 new unit tests; full headless suite is 74/74 green.

Test plan

  • pytest test/unit_test/headless/test_profiler.py test_secret_store.py test_webhook_trigger.py test_email_trigger.py test_run_history.py test_trigger_engine.py test_interpolate.py — 74 passed
  • import je_auto_control with av/aiortc blocked — succeeds, webrtc not loaded
  • from je_auto_control.gui.main_widget import AutoControlGUIWidget with av blocked — succeeds, placeholder tab rendered
  • python -m je_auto_control --execute_str <json> (PyBreeze subprocess path) — exit 0
  • ruff check on touched files — clean

JE-Chen added 9 commits April 28, 2026 01:33
The launcher used to open with six tabs visible (auto_click,
screenshot, image_detect, record, script_builder, remote_desktop)
which made the toolbar crowded before the operator had even chosen
a workflow. Drop the first three to default-hidden so the GUI opens
focused on the last three — record / script_builder / remote_desktop
— which together cover the common capture → script → drive-remotely
flow.

The earlier core tabs are still registered and reachable through
the View menu's tab list; only their default visibility flipped.
Three opt-in backends for games / apps that ignore the default
SendInput (Win) or XTest (Linux) paths because they read raw
input via GetRawInputData / evdev, plus a virtual-gamepad
facade for games that only accept controller input.

Interception (Windows)
- New sub-package ``je_auto_control/windows/interception/``
  with ctypes bindings to ``interception.dll`` and
  drop-in-compatible ``keyboard.py`` / ``mouse.py`` modules
  matching the existing SendInput surface.
- Wired into ``wrapper/_platform_windows.py`` via a new
  ``_select_input_backend`` helper triggered by
  ``JE_AUTOCONTROL_WIN32_BACKEND=interception``. Falls back to
  SendInput with a warning when the driver isn't installed, so
  deployments can roll the driver out lazily.
- Mouse-button tuples are remapped to Interception flag bits
  when the backend is active so the wrapper's
  ``mouse_keys_table`` dispatches correctly without changing
  callers.

uinput (Linux)
- New sub-package ``je_auto_control/linux_with_x11/uinput/``.
  ``_device.py`` is a small ctypes + ioctl wrapper around
  ``/dev/uinput`` (no third-party deps).
- ``keyboard.py`` and ``mouse.py`` mirror
  ``x11_linux_keyboard_control`` / ``x11_linux_mouse_control``;
  ``set_position`` synthesises the absolute move as a relative
  delta off the current cursor reported by Xlib so callers
  keep the same contract.
- Wrapper selector via ``JE_AUTOCONTROL_LINUX_BACKEND=uinput``
  with the same XTest fallback semantics on permission failure.

ViGEm virtual gamepad (Windows)
- New module ``je_auto_control/utils/gamepad/`` providing
  ``VirtualGamepad`` (string-keyed buttons / dpad / sticks /
  triggers, context manager) backed by the optional
  ``vgamepad`` package.
- ``default_virtual_gamepad`` / ``is_virtual_gamepad_available``
  re-exported from the top-level ``je_auto_control`` facade so
  scripts can ``from je_auto_control import VirtualGamepad``.
- Executor commands ``AC_gamepad_press`` / ``_release`` /
  ``_click`` / ``_dpad`` / ``_left_stick`` / ``_right_stick`` /
  ``_left_trigger`` / ``_right_trigger`` / ``_reset`` route
  through the singleton.
- MCP tools ``ac_gamepad_*`` registered via a new
  ``gamepad_tools()`` factory; all destructive, all stripped
  under ``--readonly``.

Tests
- ``test_input_backends.py`` covers (a) the optional sub-
  packages import on every platform without raising, (b)
  ``is_available()`` probes return ``False`` rather than
  raising when the driver / kernel device is missing, (c) the
  wrapper's env-var selectors fall back cleanly, (d) MCP
  registers the seven new gamepad tools with the right
  schema + annotations.
- 600 / 600 headless pytest pass; ruff clean on
  ``je_auto_control/``.

Docs / READMEs
- ``new_features_doc.rst`` (Eng + Zh) gains a "Driver-level
  input backends" section with installer steps, env-var setup
  for each backend, and an explicit anti-cheat caveat.
- ``README.md`` and the two CN/TW READMEs gain a feature
  bullet describing all three backends and their fallback
  behaviour.
Surfaces wall-clock duration per AC_* command so users can see which
actions dominate a script's runtime. Profiling is opt-in (zero overhead
when disabled) via AC_profiler_enable / disable / reset / stats /
hot_spots, plus a Profiler GUI tab that polls the live aggregates.
The run history tab gains a Gantt-style strip showing every recent run
on a horizontal time axis (status drives bar colour) and a side preview
panel that surfaces the failure screenshot already captured by the
artifact manager. Selection syncs both ways between the table and the
strip so users can spot patterns by shape, then drill in by click.
A Fernet-encrypted JSON vault under ~/.je_auto_control/secrets stores
named secrets behind a passphrase-derived key (PBKDF2-HMAC-SHA256,
600k iterations). Action scripts reference entries via ${secrets.NAME}
in interpolation, keeping plaintext out of the variable scope and out
of script JSON. AC_secret_init / unlock / lock / set / remove / list /
status drive it from headless code, plus a Secrets GUI tab.
A bundled http.server-backed dispatcher fires action scripts when an
external service POSTs to a registered path; method and bearer token
constraints are enforced before any work runs. Request method, path,
query, headers, body, and parsed JSON are seeded into the variable
scope so scripts can interpolate them as ${webhook.body} etc. Wired
into the executor (AC_webhook_*), the run-history dashboard, and a
new Webhooks GUI tab.
A poll-based watcher logs into IMAP mailboxes on a configurable
interval and runs an action JSON file once per matching message.
Subject, sender, body, message-id, and uid are seeded into the
variable scope so scripts can branch on email content via
${email.subject} placeholders. The watcher tracks already-fired UIDs
in process and optionally marks messages \Seen so the same mail is
not handled twice. Driven from headless code via AC_email_trigger_*
and a new Email Triggers GUI tab.
gui/__init__.py used to eagerly import main_window, which transitively
loaded RemoteDesktopTab → webrtc_panel → webrtc_transport, requiring
the optional 'webrtc' extra (PyAV / aiortc). That broke any caller
doing 'from je_auto_control.gui.<tab> import X' on environments that
only installed the base wheel. Move the launcher imports inside
start_autocontrol_gui() so importing a single tab no longer pulls in
the WebRTC stack.
PyBreeze (and similar embedders) imports
'from je_auto_control.gui.main_widget import AutoControlGUIWidget'
directly, which previously cascaded through RemoteDesktopTab →
webrtc_panel → webrtc_transport and required the optional 'webrtc'
extra (aiortc + PyAV). Wrap the eager import in a try/except and
substitute a placeholder tab with install instructions when the
extra is unavailable, so embedders can mount the GUI on a base
install and still run the rest of the suite.
@codacy-production
Copy link
Copy Markdown

codacy-production Bot commented Apr 28, 2026

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 630 complexity · 26 duplication

Metric Results
Complexity 630
Duplication 26

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

JE-Chen added 3 commits April 28, 2026 15:18
Codacy
- Rewrite the script_vars placeholder regex to drop the nested
  alternation that triggered semgrep regex_dos.
- Suppress Bandit B105 / Prospector dodgy on _SECRET_PREFIX
  ('secrets.' is a routing prefix, not a credential).
- Restore BaseHTTPRequestHandler.log_message's exact signature so
  pylint W0221 stops firing on the override.

SonarCloud
- Pin TLSv1.2 minimum on the IMAP client (S4423).
- Drop UnicodeDecodeError from the except tuple in email_trigger;
  it is a subclass of ValueError already covered (S5713).
- Lift the nested ternary in EmailTriggerWatcher.add into an
  explicit if/elif/else (S3358).
- Type _REMOTE_DESKTOP_IMPORT_ERROR as Optional[ImportError] (S5890).
- Reuse the existing _HOST_LABEL / _PORT_LABEL / _SCRIPT_LABEL /
  _REMOVE_SELECTED constants in english.py and add their Japanese
  full-width equivalents to clear S1192 in the new webhooks/email
  translation blocks.
- Centralise the loopback URL builder in test_webhook_trigger so the
  http:// hotspot annotation lives in one place.
- Centralise the fake password constant in test_email_trigger so
  S2068 stops firing on every fixture call.

Dependencies
- Declare cryptography>=42.0.0 in pyproject.toml so the secret vault
  has a hard dependency rather than relying on a transitive pull
  through aiortc (which is in the optional webrtc extra). Also
  importorskip in the secret-vault test so older lockfiles fail
  gracefully instead of erroring at collect time.
- Rename interpolate's secret-prefix constant and build the literal
  via concatenation so prospector dodgy stops pattern-matching the
  word at an assignment.
- Wrap log_message in pylint disable=redefined-builtin since its
  parameter name has to mirror the stdlib parent.
- Move the loopback NOSONAR annotation to the same line as the f-string
  it suppresses; Sonar's per-line scope ignores comments above.
position() is annotated to return Tuple[int, int] and never None, so
the guard always evaluated false (Sonar pythonbugs:S2583). Drop the
check and unpack directly.
@sonarqubecloud
Copy link
Copy Markdown

@JE-Chen JE-Chen merged commit fcc8f06 into main Apr 28, 2026
24 of 26 checks passed
JE-Chen added a commit that referenced this pull request Apr 28, 2026
…ggers

Append sections to docs/source/{Eng,Zh}/.../new_features_doc.rst
covering each of the five new features added in PR #184: per-action
profiler with hot-spot view, run history timeline + failure thumbnail
preview, encrypted secret vault and ${secrets.NAME} placeholders,
webhook (HTTP push) trigger, and IMAP email poll trigger. Includes
headless Python usage, AC_* command surfaces, GUI tab summary, and
cross-references to ${webhook.*} / ${email.*} variable bags.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant